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ABSTRACT 



This paper describes the implementation and support of a 
data-driven decision-making evaluation system developed by the San Francisco 
(California) school district to assess programs within the district aimed at 
improving achievement for students of color, especially African American and 
Latino students. In 1982, a Consent Decree was established to meet the goals 
of desegregation of San Francisco's schools. Under the Consent Decree, the 
school district receives state money to implement programs for targeted 
students. An evaluation model was developed to assess these programs. The 
model focuses on: (1) project description; (2) professional development; (3) 

qualitative evaluation; (4) quantitative evaluation; and (5) budget 
considerations. The model can be used with a range of programs while still 
providing specific information about key areas of concern to school district 
administrators and the Board of Education. Participatory evaluation is a 
feature of the model, and outcome evaluation is an essential aspect. The 
experiences of the San Francisco school district with this approach have 
shown that it is possible to develop and implement an evaluation approach 
that can be used for many projects. (Contains 9 references.) (SLD) 
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Assessing Achievement Effects of Desegregation Programs: 
The Role of a Program Evaluation and Research Department 



Introduction: 

In recent years, there has been much attention to the relationship between achievement 
and desegregation programs in large urban districts in the United States. As pressure has come 
from many sectors in communities to end desegregation, there is evidence that ending court- 
ordered desegregation can have deleterious effects on smdent achievement for low-income 
minority students (Orfield et al., 1996). In Norfolk, Virginia, one year after the courts declared 
the schools sufficiently integrated, test scores fell for both white and African American smdents 
(Ipka. 1993). At the same time, some researchers have questioned whether desegregation 
programs are effective in helping improve student achievement, suggesting that rising economic" 
opportunities for African American students explain better rising test scores for African 
Americans in desegregated districts (Armor, 1992). In the current political context, in which 
desegregation programs are being challenged in court and being debated among researchers, it is 
critical to develop systems within districts to measure the effects of programs (both individual 
and district- wide) aimed at improving educational opportunities for smdents of color. 

There have been only a few smdies that have examined the effects of programs that are 
funded within districts to address the academic needs of smdents targeted by desegregation and 
integration efforts. These smdies. moreover, have been n pically limited to smdies of a single 

program in a district (e.g., Stanford & Bellott, 1982). 

The objective of the current smdy is to describe the implementation and support of a 
data-driven decision-making evaluation system developed by one district to assess a rnultimde of 
programs within the district aimed at improving achievement for smdents of color, particularly 
African American and Latino smdents. 

Institution of the Consent Decree in San Francisco 

San Francisco smdents have attained achievement levels above the national average in 
spite of both the fact that we are educating increasing numbers of children effected by poverty 
and that the State education funding is markedly below average- California ranked 22nd among 
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the states in 1978 and has since plummeted to 41st place among the states in the amount spent 
per child (Digest of Educational Statistics). This resilience is in part due to special programs that 
have been instituted, through the demand of the courts, that minimize segregation and help 
students succeed in school. 

In 1982 in District Federal Court, the NAACP and SFUSD came to an agreement about a 
Consent Decree to meet the goals of desegregation of San Francisco’s schools. Far from being a 
supplemental program for a handful of schools, the Consent Decree has had a systemic influence 
on the entire School District's belief and expectation that all children are capable of achieving at 
high levels regardless of ethnicity, language, or family income level. While much remains to be 
done to eliminate gaps in achievement and other outcomes, the Consent Decree has made great 
progress toward creating educational equity and excellence for all of our students. In the words 
of a 1992 report by a Court-appointed Committee of Experts: 

“[The Consent Decree] has made possible one of the most extensive 
educational reform efforts that have been carried out in the last 
generation in an urban school district. 

The goals of the Consent Decree have been twofold: 

1 . Continued and accelerated efforts to achieve academic excellence for all students, 
with a particular focus on African American and Latino youngsters, the lowest 
performing groups 

2. Elimination of racial/ethnic segregation and identifiability in SFUSD schools, 
classrooms and programs 

In linking the goal of academic achievement with that of desegregation, San Francisco's Consent 
Decree stands out as unique from most other desegregation court orders or consent decrees in the 
U.S. The Consent Decree's process of reshaping the educational culture and expectations of the 
San Francisco Unified School District has benefited not only African Amencan and Latino 
students, but aU of the District's students and schools, and has done so against powerful societal 

odds. 




Supporting Programmatic Efforts by Accounting for Success 

Through the Consent Decree, the District receives $32 million from the state annually to 
implement programs for targeted students that support the Consent Decree’s goals, which are to 
integrate all public schools in San Francisco and to increase academic excellence of targeted 
students. Court monitors and a committee of experts supervised the achievement of the two 
goals. The monies were distributed to schools, administrative staff and to programs that served 
the two goals. It was seen that extensive reports were produced accounting for the success of the 
schools, but no evaluation of the programs was done. The Board of Education Commissioners, 
state monitors, and the district’s Committee of Expens consistently stressed that the 
effectiveness of educational programs on student achievement was not being assessed. 

In July, 1994. incoming Assistant Superintendent for Consent Decree programs. Dr. Tony 
Anderson, selected the top ten programs (identified because their costs exceeded $100,000) for 
evaluation. These programs included well-known names like Reading Recovery, Success For 
All, Step to College as well as a host of locally-sponsored ones. The Research, Planning and 
Evaluation (RPE) vvas called upon to determine a plan for evaluating the performance of these 
programs during the 1993/94 academic year. Action taken by the RPE involved establishing a 

model for program evaluation. 

RPE developed an evaluation model for the District which may be used with a range of projects 
while still providing specific information about areas of key concern to District administrators and the 
Board of Education. The model focused on the five areas described below; 

. Proi.^ct description— this encompassed descriptions of involved populations, project goals 
and objectives, implementation framework and strategies, student selection strategies, and 
indicators or benchmarks that project administrators intended on using to judge project 

impacts. 

, Professional development — this was an examination of the project efforts to engage in 
continuous improvement, to make schools self sufficient or to instimtionali 2 e the program. 

. A qnaiitative evaluation— this component included surveys/interviews of stakeholder groups 
(school staff, students and parents) and on-going site visits to document implementation. 



The Role of a PE&R Department 



4 



Khaima, R., Trousdale, D., Penuel, B. 



* A quantitative evaluation — this component was tied to the specific goals and objectives of 
the District, used experimental and quasi -experimental designs, with the data maintained by 
the District about such areas as gp^, attendance, suspensions, and standardized test scores to 
determine effectiveness. 

• A budget— this component involved a cost benefit'analysis detailing whether the level of 
expenditures were both consistent with the level of effort the project demanded. Further it 
tracked direct verses indirect cost of services to smdents. Similar programs were compared 
on their per student cost. 

During the 1994-95 school year the above evaluation model was designed to accommodate the 
diverse project mix and the multiple areas where projects were anticipated to impact students. District 
staff, the Superintendent, and the Board of Education were encouraged by the results from the ten 
evaluations. We found a broad evaluation approach could be used which was both flexible and still 
provided valuable formative and summative information about projects. 

The evaluation model was accepted and adopted by the Board of Education. On July 13, 

1995 the Board of Education Resolution No. 55-23A2 was adopted. It stated that all 
instructional/intervention programs should have an evaluation component and that all 
evaluations will have sound educational assessment designs. Action taken by the newly 
restructured Research, Planning and Evaluation (RPE) Department involved reviewing the 
evaluation component of all board resolutions. Training workshops on the subject of evaluation 
was provided to all administrators. 
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Development and Implementation of the Evaluation Model 

The evaluation model could be used with a range of programs while stiU providing 
specific information about areeis of key concern to District administrators and the Board of 
Education. The first step in this process was development of a plan for identification of the 
educational interventions that would require a more comprehensive evaluation. This was 
necessary to account for the vast variety of interventions that take place in the District. A rubric 
was designed to help admini strators and external agencies link the objectives of the intervention 
to its outcomes. The rubric was defined in terms of levels with each level directly corresponding 
to the appropriate type of evaluation that should take place. It then became a requirement that 
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administrators specify their goals and define the level of evaluation required for ever>' resolution 
request that goes before the Board of Education. The evaluation discussed in this paper is 
reserved for Level IV evaluations. Intert'entions whose activities fall under Levels 1 . 2, and 3 
are required to file in-house evaluation reports so that their administrators are accountable for 
faithful implementation of project activities. 



Table 1: Description of Evaluation Levels 



If the objective of the activities or 
Services is-** 


The level of evaluation that 
will be reduired is... 


Type, of Evaluation , 


to complete a task or produce a product 
or to expose panicipants to the activities 
or ser\'ices 

to expose participants to the activities or 
services and to collect feedback 
regarding their perceptions 


Level I 
Level II 


A record that the task or • 
activity has taken place 

A post survey that collects 
feedback which will inform 
upon the delivery of the 
intervention 


to change the attitude or opinion of the 
panicipants 


Level III 


A pre- and post survey that 
assesses whether attitudes or 
opinions have changed 


to change the behavior of the 
participants 


Level rv 


A full evaluation that assesses 
appropriate implementation of 
the intervention and its 
effectiveness to change 
soecified behavioral outcome 



The Consent Decree provides support for a number of effective programs throughout the 
District which have been identified as needing Level IV evaluations. These programs target the 
academic success of a diverse audience of students throughout the District and involve a number 
of stakeholders in the educational process. From direct intervention with targeted groups of 
students to district-wide support programs, professional development programs to parental and 
community involvement, these programs work in concert towards academic success ot all 
students throughout the district. On the following page. Table 2 provides a bnei descnpnon of 
goals, implementation and outcomes of some of the educational interventions funded through the 

Consent Decree that were evaluated by RPE. 




BEST COPY AVAILABLE 



7 



The Role of a PE&R Department 



6 



Khanna, R.. Trousdale, D., Penuel, B. 



Table 2: Description of Some of the Consent Decree Funded Programs Evaluated 



Using the RPE Evaluation Model 



Program Goals 


Program Implementation 


Program Outcomes 


Promote scholastic 
achievement of AA/Latino 
students 

1 


provides accelerated 
academic courses on a 
college campus, 
supplemented with 
educational enrichment and 
support structures to ensure 
student success 


The number of students 
taking advanced placement 
and honors classes 
increased among those 
participating in the 
program, and students 
confirm that the program 
gave a glimpse of university 
life. 


To improve the 
performances and prospects 
of African American 
students 


A K-8 “culture rich” 
language and mathematics 
literacy intervention. The 
program supports 
professional development, 
curriculum, instructional 
improvement in language 
arts and mathematics, and 
community support to reach 
these students 


Students whose teachers 
were professionally 
developed showed growth 
on a standardized test in 
math. Participants said that 
the program was successful 
with targeted groups of 
smdents. 


A youth development 
program that empowers 
smdents with the skills they 
need to be effective leaders 
who service their peers, 
schools, and communities 


Implemented in all 17 of 
San Francisco’s middle 
schools and in eight high 
schools. Last year, the 
program trained over. 800 
“peer educators,” who in 
turn served over 1 0,000 
other students. 


Program participants are 
more likely to have better 
grades.-stay in school, stay 
out of gangs, stay drug free 
and involve themselves in 
alternative positive 
activities. 


Intensive reading program 
aimed at organizing 
resources at the school site 
to ensure that virtually 
every student will reach 
third grade with a 
foundation of basic skills 
necessary for future 
academic success 


Program includes staff 
development, non-graded 
reading blocks, one-on-one 
reading tutors, specialized 
reading material and a 
family support team 
induing social services as 
needed 


Almost 1 000 smdents in 
three schools benefited 
from the program in 3 
different schools. 
Participating smdents 
showed more than a year’s 
gain on a standardized 
reading test. 
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The Level IV evalualion model was laier revisited and extended to accommodate the 
various needs of all stakeholders involved. To evaluate the effectiveness of these programs, two 
eontponents were used-a process component and an outcome component. As the evaluation 
procedure began to take shape and improve, it became clear that if programs were going to use 
results effectively for program improvement, evalualion could not be seen as threatening. 

Therefore RPE currently employs an approach to evaluation that is participatory (Cousins & 

Earl. 1992), utilization-focused (Patton, 1986; 1994), and integrated with processes of 
continuous’improvement and program planning (Fetterman, Kaftarian, & Wandersman, 1 996). 
Moreover, the model is based on the idea that participation of program directors and coordinators 
in the evalualion process is key to ensuring that program planners and managers use the 
evaluation data to support decision-making. The tnvolvement of program directors and 
-caordmators has the potential to encourage program staff to think more systematically about the 
relationship between program activities and objectives. Such systematic reflection would be 
aimed at building a -‘culture of learning” (Panon, 1997, p. 147) to lead to continuous program 

improvement. 

The RPE employs participaTory evaluation in several ways. For one, the department mes 
to work closely with program directors to help insure that inter^'ention activities are closely 
connected with desired outcomes. In the third year of instituting the evaluation model, RPE 
began to conduct on-going discussions and meetings with program staff that focus on making 
connections made between activities and distant outcomes. Below is an excerpt of a matnx 
created bv proeram staff in conjunction with the evaluator. The staff was asked to identify all 
activities'that would be a pan of the program. They were then asked to identify the desired 
outcomes and their assumptions underlying why they believed panicipauon in the activity would 



lead to the outcomes they had specified. The creation of this matnx was a crucial step m 
insuring that programmatic design would lead to desired outcomes. 



Activity 

Tutoring 



assumptions 
Tutoring vr&nsters skilU 



Nearby outcomes 

^ Less disruptive in 
class 



Distant Outcomes 



More lime nn 

► 



Improve 
literacy and 
math skills 
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The RPE works with program directors to help specify goals, identify benchmarks, and 
generate appropriate measures of program success. In this regard, both the evaluator and the 
program administrators have a mumally defined understanding of what they are being held 
accountable to achieve. TTie creation of survey instruments is also done with the input of the 
program staff. This insures that the questions asked of the program participants will generate 
da ta that is both useful for making program improvements and accounting for successful 
program implementation and outcomes. Results of data collected throughout the school year (i.e. 
site visit observations, mid-year surveys, semester gpa’s or attendance rates) are often shared 
with program administrators on an on-going basis to further insure that evaluation is beneficial 
and participatory rather than pumtive. 

The outcome model has also continuously improved over the years and is designed to 
communicate implementation and achievement results to school commumty stakeholders and to 
hold programs accountable for achieving their set goals. The layout of the reports is as follows: 

» Program Design which provides information regarding program history (how the 
program was developed and has evolved over the years), objectives, staff 
qualifications and training, a theoretical rationale for why the program is needed and 
an educational rationale for how the program is designed (description of published 
research that supports the program model). 

• Program Implementation which includes descriptions of process methodology such as 
the process for identification of participants, activities conducted, detailed accounts of 
implementation procedures, and perceived barriers and supports provided by 
stakeholders (school/program staff, students, and parents). 

• Program Outcomes gives us the findings of the program implementation. It begins 
with a description of the survey design process, sampling methods and analysis 
methods. Also, the quantitative design: instruments used, kind of design (e.g. pre- 
post, comparison), and statistical tests to be used are described. The key findings 
from surveys are synthesized and discussed in this section. Survey results include 
discussions that focus around common themes, discrepancies and perceived success 
of the program. Analysis of data in relation to the key outcomes is also done in 
accordance with a quantitative research design. Lastly, suggestions for improvement 

10 
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of the program are provided keeping in mind the goal that the program needs to be 
more effective in achieving its outcomes. 

The audience for these reports is the program director, district administrators, the 
Superintendent, the Board of Education, the state monitors and the committee of expens. 

Program Accountability; Decision-Making Based on Evidence 

Communication of results and decision-making has also improved over the years. 
Evaluation reports that were produced during the first year merely confirmed success and 
practice of the evaluation model. In its second year of implementation, procedural and 
evaluative recommendations were made to program directors and district administrators in 
charge of the programs. 

During the third year of practice. 1996-97. the department was able to ha\ e major impact 
in terms of decisions with regards to Consent Degree programs (see Table 3). L'sing both 
formative and summative information from the evaluation reports, three types of decisions 
emerged. Programs that did not show improvement over the three-year period were given a “red 
light” or were discontinued. This was equivalent to 27% (4/ 1 5) of the Consent Decree 
programs. Programs that had the purpose and planning components in place but had not yet 
yielded success with regards to implementation were given a “yellow light” and another year to 
vyork on their barriers. These were equivalent to 43% (7/15) of Consent Decree programs. 
Twenty percent ( 3 / 15 ) of programs received a “Green light” for demonstrating .continual success 
with smdents. Table 3 on the following page is an excerpt of the program summar>- matrix with 
regards to our decisions at the end of every school year. 
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Table 3: Excerpt froro Program Decision Summary Matrix 



Funding 


Program 

Description 


Decision 


Reasons 


$157,885 


Program 
designed to 
help students 
achieve 
Chinese oral 
and vvritten 
language 
proficiency 






Green 


1 . Grant funded 

2. Unique, before and after school 
Cantonese/Mandarin language 
program 

3. Students and parents rated the 
program as very good 


$100,000 


A Latino 
retention 
program for 
high school 
students 


Red 






1 . Serves very few smdents 

2. No significant gains on GPA or 
CTBS 

3. Students served were not at-risk 
(CTBS scores were at the national 
average, GPA was over 2.0, and 
attendance over 80%.) 


$50,000 


An after- 
school 
program for 
high achieving 
African 
American 
smdents. 




YcUow 




1 . One years growth on CTBS 

2. No difference from comparison group 

3. Unique, serves as an after school site 
for African American students 


$185,000 


Program that 
. offers tutorial 
ser\ices to at- 
risk students 
with GPA<2.0 


Red 






1. Significant decrease in GPA 

2. Significant decrease in attendance 

3. Graduation rate 65.6% 




Early 

intervention 

Reading 

Program 




YfeUow 




1 . Significantly lower performance than 
a random sample of similar grade 
students, no difference between 
successful and unsuccessful (as per 
program measures) AA/L students on 
the CTBS reading sub-test 

2. High attrition rate, no follow-up for 
unsuccessful students 

3. Unique, serves a population that 
needs the extra assistance 
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ImpUcatioDS 



The experience of RPE and District staff has shown that it is possible to develop and 
implement an evaluation approach, which is usable in a large school district, which operates 
many projects. Several benefits have emerged from our work to date. First, budgetary decisions 
are being made in the district based on the evaluation results. Second, administrators have a 
clear set of expectations that must be considered when implementing and evaluating before-, 
after-, and in-school programs. Third, evidence from current year evaluations suggests that 
projects are becoming more focused as the level of accountability to the Consent Decree office 
and District is increasing. Use of a s>-stematic approach to evaluation has highlighted 
weaknesses in some programs. Pooriy designed projects are being forced to develop more 
coherent frameworks and activities or risk losing future funding. The Distnct is benefitmg from 
having a common framework for analyzing programs by providing administrators with more data 
to make informed decisions about the costs and benefits of alternative approaches to meeting the 

same goals. 
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